In this paper, we propose to exploit unlabeled videos for semi-supervised learning of optical flow with a Generative Adversarial Network.

Summary: The paper presents a semi-supervised approach to learning optical flow using a generative adversarial network (GAN) on flow warp errors. Rather than using a handcrafted loss (e.g., deviation of brightness constancy deviation from smoothness) the paper explores the use of a GAN applied to flow warp errors. Strengths: novel semi-supervised approach to learning; some concerns on the novelty in the light of [21] generally written well Weaknesses: - some key evaluations missing Comments: Supervised (e.g., [8]) and unsupervised (e.g., [39]) approaches to optical flow prediction have previously been investigated, the type of semi-supervised supervision proposed here appears novel. The main contribution is in the introduction of an adversarial loss for training rather than the particulars of the flow prediction architecture. As discussed in Sec. 2, [21] also proposes an adversarial scheme.

Neural Information Processing Systems http://nips.cc/

Convolutional neural networks (CNNs) have recently been applied to the optical flow estimation problem. As training the CNNs requires sufficiently large ground truth training data, existing approaches resort to synthetic, unrealistic datasets. On the other hand, unsupervised methods are capable of leveraging real-world videos for training where the ground truth flow fields are not available. These methods, however, rely on the fundamental assumptions of brightness constancy and spatial smoothness priors which do not hold near motion boundaries. In this paper, we propose to exploit unlabeled videos for semi-supervised learning of optical flow with a Generative Adversarial Network. Our key insight is that the adversarial loss can capture the structural patterns of flow warp errors without making explicit assumptions. Extensive experiments on benchmark datasets demonstrate that the proposed semi-supervised algorithm performs favorably against purely supervised and semi-supervised learning schemes.